Rewards Analytics and Distribution Dashboard for Quantification Review

This goal of this document is to offer an easy way to process the outputs of the praise reward system and perform an analysis of the resulting token reward distribution. It should be considered a work-in-progress.

Out[6]:

Distribution report for round-12

  • This period covers praise given between 2022-05-16 and 2022-05-28.
  • We allocated a total of 1876 TEC tokens for rewards.
  • Duplicate praise received a weighting of 0.1 the value of the original praise.
  • We assigned 4 quantifiers per praise instance.
  • Praise receiver names were not hidden behind pseudonyms during quantification

Praise Data Visualization

Rating distribution

Since praise gets valued on a scale, we can take a look at how often each value of the scale gets assigned by quantifiers. Note: how to process the duplicate scores? For now, just delete them.

TOP10 highest rating praise

Praise score average: 103.06
FROM divine_comedian#5493 TO Mount Manu#3530,reason:
nan

Praise score average: 102.75
FROM Tam2140#9361 TO wslyvh#1059,reason:
nan

Praise score average: 84.67
FROM mattyjee#8621 TO iviangita#3204,reason:
nan

Praise score average: 75.25
FROM divine_comedian#5493 TO Mert Ozd#6679,reason:
nan

Praise score average: 61.5
FROM r33pich33p#6906 TO ALOYSIOUS#3049,reason:
nan

Praise score average: 53.67
FROM Vyvy-vi#5040 TO zhiwei#1356,reason:
nan

Praise score average: 48.0
FROM pat.zip (TE Academy)#5266 TO EFRA#7972,reason:
nan

Praise score average: 48.0
FROM Juankbell#7458 TO Zeptimus#3359,reason:
nan

Praise score average: 48.0
FROM GideonRo#3175 TO enti#1546,reason:
nan

Praise score average: 46.5
FROM r33pich33p#6906 TO GideonRo#3175,reason:
nan

Praise Reward Distribution

We can now take a look at the distribution of the received praise rewards. You can toggle the inclusion of the different sources by clicking on the legend.

Praise Giving Distribution

We can also take a look at the distribution of the people giving praise.

Praise Flows

Now for something more fun: let's surface the top "praise flows" from the data. Thanks to @inventandchill for this awesome visualization! On one side we have the top 20 praise givers separately (modifiable by changing the variable n_senders), on the other the top 25 receivers (modifiable by changing the variable n_receivers). The people outside the selection get aggregated into the "REST FROM" and "REST TO" categories.

Out[14]:

Quantifier Data

Let's take a closer look at the quantification process and and see if we can spot any problems:

Praise Outliers

To aid the revision process, we highlight disagreements between quantifiers.

Outliers sort by spreads

Here we generate a table which sorts the praise by the size of the spread between the highest and lowest scores. It gives us an overview of the spread distribution.

For an exhaustive list, take a look at the exported file "praise_outliers.csv" .

This is a visual aid. ATTENTION! If there are several praise instances with similar spread and quant score, all but one end up "hidden" on the chart.

Praise score by quantifier -- outliers among the quantifiers?

Let's see the range of praise scores every quantifier gave to see the behavior difference of quantifiers.

To interpret the box plot:

  • Bottom horizontal line of box plot is minimum value

  • First horizontal line of rectangle shape of box plot is First quartile or 25%

  • Second horizontal line of rectangle shape of box plot is Second quartile or 50% or median.

  • Third horizontal line of rectangle shape of box plot is third quartile or 75%

  • Top horizontal line of rectangle shape of box plot is maximum value.

score displacement: tendency to under/over-scoring?

scoring correlation: how similiar am i scoring with others?

Agreement on duplication

Among 342 praises, 32 (9.28%) do not agree on duplication

Praise instances with disagreements in duplication are collected in 'results/duplication_examination.csv'. To compare, look at the last 4 columns: 'DUPLICATE MSG 1/2/3' and 'ORIGINAL MSG'.

Agreement on dismissal

Among 342 praises, 27 (7.83%) do not agree on dismissal

Praise instances with disagreements in dismissal are collected in'results/dismissal_disaggreed.csv'. You can further look into who dismissed and who did not.